Apparatus, system, and method of processing image data to be relayed

ABSTRACT

A relay apparatus for transferring image data transmitted from a transmission source to a transmission destination, including circuitry to: receive video image data from a first apparatus being the transmission source; determine whether a second apparatus being a transmission destination is capable of reproducing the video image data received from the first apparatus; based on a determination that the second apparatus is not capable of reproducing the video image data received from the first apparatus, apply image processing to the video image data to generate image data compatible with the second apparatus; and transfer the image data compatible with the second apparatus, to the second apparatus.

CROSS-REFERENCE TO RELATED APPLICATIONS

This patent application is based on and claims priority pursuant to 35U.S.C. § 119(a) to Japanese Patent Application Nos. 2018-225918, filedon Nov. 30, 2018, and 2019-209599, filed on Nov. 20, 2019, in the JapanPatent Office, the entire disclosure of which is hereby incorporated byreference herein.

BACKGROUND Technical Field

The present invention relates to an apparatus, system, and method ofprocessing image data to be relayed.

Description of the Related Art

Recently, streaming is used to continuously transmit data, such as audioand video, from one side to other side. Through streaming, image datacaptured at a communication terminal on a sender side is transmitted toa communication terminal on a receiver side, and processed in real timefor display on the receiver side.

SUMMARY

Example embodiments include a relay apparatus for transferring imagedata transmitted from a transmission source to a transmissiondestination, the relay apparatus comprising circuitry to: receive videoimage data from a first apparatus being the transmission source;determine whether a second apparatus being a transmission destination iscapable of reproducing the video image data received from the firstapparatus; based on a determination that the second apparatus is notcapable of reproducing the video image data received from the firstapparatus, apply image processing to the video image data to generateimage data compatible with the second apparatus; and transfer the imagedata compatible with the second apparatus, to the second apparatus.

Example embodiments include a system for processing image data to berelayed between a transmission source and a transmission destination,the system comprising circuitry configured to: receive video image datafrom a first apparatus being the transmission source; determine whethera second apparatus being a transmission destination is capable ofreproducing the video image data received from the first apparatus;based on a determination that the second apparatus is not capable ofreproducing the video image data received from the first apparatus,apply image processing to the video image data to generate image datacompatible with the second apparatus; and transfer the image datacompatible with the second apparatus, to the second apparatus.

Example embodiments include a method of processing image data to berelayed between a transmission source and a transmission destination,the method comprising: receiving video image data from a first apparatusbeing the transmission source; determining whether a second apparatusbeing a transmission destination is capable of reproducing the videoimage data received from the first apparatus; based on a determinationthat the second apparatus is not capable of reproducing the video imagedata received from the first apparatus, applying image processing to thevideo image data to generate image data compatible with the secondapparatus; and transferring the image data compatible with the secondapparatus, to the second apparatus.

BRIEF DESCRIPTION I/F THE SEVERAL VIEWS I/F THE DRAWINGS

A more complete appreciation of the disclosure and many of the attendantadvantages and features thereof can be readily obtained and understoodfrom the following detailed description with reference to theaccompanying drawings, wherein:

FIG. 1 is a schematic diagram illustrating a configuration of acommunication system according to an embodiment;

FIG. 2 is a schematic diagram illustrating a hardware configuration of areal-time data processing terminal, according to an embodiment;

FIGS. 3A and 3B (FIG. 3) are schematic diagrams each illustrating ahardware configuration of an imaging unit, according to an embodiment;

FIG. 4A is a view illustrating a hemispherical image (front side)captured by the imaging unit;

FIG. 4B is a view illustrating a hemispherical image (back side)captured by the imaging unit;

FIG. 4C is a view illustrating an image in equirectangular projection;

FIG. 5A and FIG. 5B are views respectively illustrating the image inequirectangular projection covering a surface of a sphere, and aspherical image, according to an embodiment;

FIG. 6 is a view illustrating positions of a virtual camera IC and apredetermined area T in a case in which the spherical image isrepresented as a surface area of a three-dimensional solid sphere.

FIGS. 7A and 7B are respectively a perspective view of FIG. 6, and aview illustrating an image of the predetermined area on a display,according to an embodiment;

FIG. 8 is a view illustrating a relation between the predetermined-areainformation and the image of the predetermined area;

FIG. 9 is a schematic diagram illustrating a hardware configuration ofthe terminal data processing device, according to the embodiment.

FIG. 10 is a schematic block diagram illustrating a hardwareconfiguration of the video conference terminal, according to theembodiment;

FIG. 11 is a schematic block diagram illustrating a hardwareconfiguration of the electronic whiteboard, according to the embodiment.

FIG. 12 is a schematic block diagram illustrating a hardwareconfiguration of a communication management server and a relay device ofthe communication system of FIG. 1;

FIGS. 13A and 13B (FIG. 13) are a schematic block diagram illustrating afunctional configuration of the communication system according to theembodiment;

FIG. 14A is a conceptual diagram illustrating an example communicationmanagement table;

FIG. 14B is a conceptual diagram illustrating an example oftransmittable image type management table;

FIG. 14C is a conceptual diagram illustrating an example of reproducibleimage type management table;

FIG. 14D is a conceptual diagram illustrating an example transfer imagetype management table;

FIG. 15 is a data sequence diagram illustrating example operation oftransmitting and receiving image data while determining a type of imagedata to be transmitted, according to an embodiment;

FIG. 16 is a flowchart illustrating operation of determiningtransmission of image data according to an embodiment;

FIG. 17 is a flowchart illustrating operation of determining a type ofimage data, according to an embodiment; and

FIG. 18 is a flowchart illustrating operation of determining a type ofimage data, according to an embodiment.

The accompanying drawings are intended to depict embodiments of thepresent invention and should not be interpreted to limit the scopethereof. The accompanying drawings are not to be considered as drawn toscale unless explicitly noted.

DETAILED DESCRIPTION

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the presentinvention. As used herein, the singular forms “a”, “an” and “the” areintended to include the plural forms as well, unless the context clearlyindicates otherwise.

In describing embodiments illustrated in the drawings, specificterminology is employed for the sake of clarity. However, the disclosureof this specification is not intended to be limited to the specificterminology so selected and it is to be understood that each specificelement includes all technical equivalents that have a similar function,operate in a similar manner, and achieve a similar result.

Configuration of Communication System

Referring to the drawings, one or more embodiments of the presentinvention are described. FIG. 1 is a schematic diagram illustrating aconfiguration of a communication system 1 according to an embodiment.

As illustrated in FIG. 1, the communication system 1 of this embodimentincludes a real-time data processing terminal 3, a terminal dataprocessing device 5, a personal computer (PC) 6, a video conferenceterminal 7, an electronic whiteboard 8, a communication managementserver 9, and a relay device 10.

The real-time data processing terminal 3 is a terminal that capturesimages in real-time to obtain real-time captured image data. Thereal-time data processing terminal 3 is detachably connected to animaging unit 40 provided with an image sensor that captures an image ofa target, such as a Complementary Metal Oxide Semiconductor (CMOS)sensor or a Charge Coupled Device (CCD) sensor. With this configuration,the real-time data processing terminal 3 digitizes an image captured bythe imaging unit 40 into captured image data.

The captured image data that the real-time data processing terminal 3obtains depends on a type of the imaging unit 4. In one embodiment, whenthe imaging unit 40 a to be described later is used, two types of imagecan be obtained including a planar video image and a planar still image.In another embodiment, when the imaging unit 40 b to be described lateris used, four types of image can be obtained including a spherical videoimage, a spherical still image, a planar video image, and a planar stillimage. The real-time data processing terminal 3 may encode the capturedimage data for output to the terminal data processing device 5. In casethe real-time data processing terminal 3 encodes the still image data,the still image data is encoded in a standard format applicable to astill image, such as Joint Photographic Experts Group (JPEG). In casethe real-time data processing terminal 3 encodes the video image data,the video image data is encoded in a standard format applicable to avideo image, such as H. 264. Other examples of standard formatapplicable to a video image include, but not limited to, VP8, VP9, VP10,and AV1.

The terminal data processing device 5, which is located closely to thereal-time data processing terminal 3, is connected to the real-time dataprocessing terminal 3 in a one-to-one correspondence, for example, by adata bus, a Universal Serial Bus (USB). The terminal data processingdevice 5 transmits the captured image data input from the real-time dataprocessing terminal 3 to the relay device 10 via the communicationnetwork 600. The communication network 600 is implemented by theintranet, the Internet, etc., which may be wired or wireless.

The real-time data processing terminal 3 and the terminal dataprocessing device 5 are connected with each other so as to togetherfunction as the image acquisition terminal 2.

The PC 6 reproduces the image data received from the relay device 10 todisplay the image on the display 608 to be described below. The PC 6 inthis embodiment is capable of processing various types of image, such asa spherical video image, a spherical still image, a planar video image,and a planar still image.

In example operation, the image acquisition terminal 2 transmits thecaptured image data to, for example, the PC 6 in real-time. For example,the image acquisition terminal 2 may be provided at a certain locationto capture an image of a human face, and send captured image data of thehuman face to a server or the PC 6. For example, at the server, thecaptured human face may be analyzed to identify a user. In anotherexample, the PC 6 may control a display to display the captured humanface, or an analysis result transmitted from the server. In either case,a data format of image data to be transmitted to the server or the PC 6is previously set, such that the PC 6 is able to reproduce the imagedata transmitted from the image acquisition terminal 2.

However, if the image data captured at the image acquisition terminal 2is to be transmitted to any device other than the server or PC 6 that ispreviously set, the image data generated at the image acquisitionterminal 2 may not always be compatible with such device. In view ofthis, in the following, the relay device 10 is provided, which iscapable of determining a type of image data that can be reproduced ateach device, and applying image processing to generate image data thatcan be reproduced at each device in the communication system 1.

Specifically, in this example illustrated in FIG. 1, the videoconference terminal 7 and the electronic whiteboard 8 are provided inaddition to the PC 6.

The video conference terminal 7 processes video data received from therelay device 10 to display video at the display 720, and processes audiodata received from the relay device 10 to output audio through thespeaker 715. The display 720 is electrically connected with the videoconference terminal 7 through the cable 730 such as a USB cable. Thevideo conference terminal 7 processes video data received from the relaydevice 10, to display video at the display 720. When a plurality ofvideo conference terminals 7 are used, video data and audio data aretransmitted or received through the relay device 10 between the videoconference terminals 7 to carry out a video conference. The videoconference terminal 7 according to the present embodiment is capable ofreproducing planar video image data, but not capable of reproducingstill image data.

The electronic whiteboard 8 reproduces the still image data transmittedfrom the relay device 10 to display a still image on the display 820.The user can also draw characters and the like on the display 820 of theelectronic whiteboard 8 using an electronic pen 8500 to be describedlater. The electronic whiteboard 8 of the present embodiment is capableof reproducing planar still image data, but not capable of reproducingvideo data.

The communication management server 9 centrally manages loginauthentication of the video conference terminal 7 and the electronicwhiteboard 8, communication states of the video conference terminal 7and the electronic whiteboard 8, and a communication state of the relaydevice 30.

The relay device 10 operates as a router, which receives image data fromthe image acquisition terminal 2 (terminal data processing device 5) viathe communication network 600, and sends the received image data to thevideo conference terminal 7 and the electronic whiteboard 8 via thecommunication network 600. As described blow, the image acquisitionterminal 2 is an example of a transmission source (that is, a sender).The PC 6, video conference terminal 7, and electronic whiteboard 8 areeach an example of a transmission destination (that is, a receiver).

[Hardware Configuration of Communication System]

Referring now to FIGS. 2 to 12, a hardware configuration of thecommunication system 1 is described according to the embodiment.

<Hardware Configuration of Real-Time Data Processing Terminal>

FIG. 2 is a schematic diagram illustrating a hardware configuration ofthe real-time data processing terminal 3, according to the embodiment.The real-time data processing terminal 3 includes a CPU 301, a ROM 302,a RAM 303, an EEPROM 304, a CMOS sensor (CMOS) 305, an acceleration andorientation sensor 306, a medium I/F 308, and a GPS receiver 309.

The CPU 301 controls entire operation of the real-time data processingterminal 3. The ROM 302 stores a control program for operating the CPU301. The RAM 303 is used as a work area for the CPU 301. The EEPROM 304reads or writes various data such as a control program for the real-timedata processing terminal 3 under control of the CPU 301. Under controlof the CPU 301, the CMOS sensor 305 captures an image of a target(mainly a blind spot of the imaging unit 40) to obtain image data. Theacceleration and orientation sensor 306 includes various sensors such asan electromagnetic compass or gyrocompass for detecting geomagnetism,and an acceleration sensor. The medium I/F 308 controls reading orwriting of data with respect to a recording medium 307 such as a flashmemory. The GPS receiver 309 receives a GPS signal from a GPS satellite.

The real-time data processing terminal 3 further includes an imagingunit I/F 313, a microphone 314, a speaker 315, an audio input/output I/F316, a display 317, an external device connection I/F 318, and a touchpanel 321.

The imaging unit I/F 313 is a circuit that controls driving of theimaging unit 40 when an external imaging unit 40 is connected to thereal-time data processing terminal 3. The microphone 314 is an exampleof audio collecting device, which is a built-in type, capable ofinputting audio under control of the CPU 401. The audio I/O I/F 316 is acircuit for inputting or outputting an audio signal to the microphone314 or from the speaker 315 under control of the CPU 301. The display317 may be a liquid crystal or organic electro luminescence (EL) displaythat displays an image of a subject, an operation icon, or the like. Theexternal device connection I/F 318 is an interface circuit that connectsthe real-time data processing terminal 3 to various external devices.The touch panel 321 is an example of input device that enables the userto input a user instruction to the real-time data processing terminal 3through touching a screen of the display 317.

The real-time data processing terminal 3 further includes a bus line310. The bus line 310 is an address bus or a data bus, whichelectrically connects the elements in FIG. 2 such as the CPU 301.

<Hardware Configuration of Imaging Unit>

FIGS. 3A and 3B are each a schematic block diagram illustrating ahardware configuration of the imaging unit 40, according to theembodiment. Specifically, FIG. 3A illustrates a hardware configurationof a monocular imaging unit 40 a, as an example of the imaging unit 40.FIG. 3B illustrates a hardware configuration of a compound eye imagingunit 40 b, as an example of the imaging unit 40. The imaging unit 40 isa generic term for a plurality of types of imaging units (imaging unit40 a, 40 b, etc.) having different number of imaging elements.

As illustrated in FIG. 3A, the imaging unit 40 a includes an imagingelement 401 a such as a CMOS or a CCD, a lens 402 a, and a connectionI/F 408 a to be electronically connected to the imaging unit I/F 313 ofthe real-time data processing terminal 3. When the imaging unit 40 a isconnected to the imaging unit I/F 313 of the real-time data processingterminal 3, the imaging element 401 a captures an image according to animaging control signal transmitted from the imaging unit I/F 313 via theconnection I/F 408 a. Accordingly, the imaging unit 40 a illustrated inFIG. 3A obtains a planar image.

As illustrated in FIG. 3B, the imaging unit 40 b includes imagingelements 401 b 1 and 401 b 2 each may be a CMOS or a CCD, lenses 402 b 1and 402 b 2, and a connection I/F 408 b to be electronically connectedto the imaging unit I/F 313 of the real-time data processing terminal 3.The lenses 402 b 1 and 402 b 2 are, for example, fish-eye lenses. Whenthe imaging unit 40 b is connected to the imaging unit I/F 313 of thereal-time data processing terminal 3, each of the imaging element 401 b1 and 401 b 2 captures an image according to an imaging control signaltransmitted from the imaging unit I/F 313 via the connection I/F 408 b,and transmits the captured image to the imaging unit I/F 313.Accordingly, a plurality of images is transmitted as captured imagedata. The imaging unit 40 b illustrated in FIG. 3B obtains a sphericalimage, which may be referred to as an equirectangular projection imageas described below.

Next, referring to FIG. 4A to FIG. 8, a description is given of anoverview of operation of generating an equirectangular projection imageEC and a spherical image CE from the images captured by the imaging unit40 b. FIG. 4A is a view illustrating a hemispherical image (front side)captured by the imaging unit 40 b. FIG. 4B is a view illustrating ahemispherical image (back side) captured by the imaging unit 40 b. FIG.4C is a view illustrating an image in equirectangular projection, whichis referred to as an “equirectangular projection image” (or equidistantcylindrical projection image) EC. FIG. 5A is a conceptual diagramillustrating an example of how the equirectangular projection image mapsto a surface of a sphere. FIG. 5B is a view illustrating the sphericalimage.

As illustrated in FIG. 4A, an image captured by the imaging element 401b 1 is a curved hemispherical image (front side) taken through the lens402 b 1. Similarly, as illustrated in FIG. 4B, an image captured by theimaging element 401 b 2 is a curved hemispherical image (back side)taken through the lens 402 b 2. The hemispherical image (front side) andthe hemispherical image (back side), which are reversed by 180-degreefrom each other, are combined by the real-time data processing terminal3. This results in generation of the equirectangular projection image ECas illustrated in FIG. 4C.

The equirectangular projection image is mapped on the sphere surfaceusing Open Graphics Library for Embedded Systems (OpenGL ES) asillustrated in FIG. 5A. This results in generation of the sphericalimage CE as illustrated in FIG. 5B. In other words, the spherical imageCE is represented as the equirectangular projection image EC, whichcorresponds to a surface facing a center of the sphere CS. It should benoted that OpenGL ES is a graphic library used for visualizingtwo-dimensional (2D) and three-dimensional (3D) data. The sphericalimage CE is either a still image or a moving image.

Since the spherical image CE is an image attached to the sphere surface,as illustrated in FIG. 5B, a part of the image may look distorted whenviewed from the user, providing a feeling of strangeness. To resolvethis strange feeling, an image of a predetermined area, which is a partof the spherical image CE, is displayed as a flat image having fewercurves. The predetermined area is, for example, a part of the sphericalimage CE that is viewable by the user. In this disclosure, the image ofthe predetermined area is referred to as a “predetermined-area image” Q.While the predetermined area can be set by default at least initially,the predetermined area may be changed according to a user operation.

Hereinafter, a description is given of displaying the predetermined-areaimage Q with reference to FIG. 6 and FIGS. 7A and 7B.

FIG. 6 is a view illustrating positions of a virtual camera IC and apredetermined area T in a case in which the spherical image isrepresented as a surface area of a three-dimensional solid sphere. Thevirtual camera IC corresponds to a position of a point of view(viewpoint) of a user who is viewing the spherical image CE representedas a surface area of the three-dimensional solid sphere CS. FIG. 7A is aperspective view of the spherical image CE illustrated in FIG. 6. FIG.7B is a view illustrating the predetermined-area image Q when displayedon a display.

In FIG. 7A, the spherical image CE illustrated in FIG. 5B is representedas a surface area of the three-dimensional solid sphere CS. Assumingthat the spherical image CE is a surface area of the solid sphere CS,the virtual camera IC is inside of the spherical image CE as illustratedin FIG. 6. The predetermined area T in the spherical image CE is animaging area of the virtual camera IC. Specifically, the predeterminedarea T is specified by predetermined-area information indicating animaging direction and an angle of view of the virtual camera IC in athree-dimensional virtual space containing the spherical image CE.

The predetermined-area image Q, which is an image of the predeterminedarea T illustrated in FIG. 7A, is displayed on a display as an image ofan imaging area of the virtual camera IC, as illustrated in FIG. 7B.FIG. 7B illustrates the predetermined-area image Q represented by thepredetermined-area information that is set by default. The followingexplains the position of the virtual camera IC, using an imagingdirection (ea, aa) and an angle of view α of the virtual camera IC.

Referring to FIG. 8, a relation between the predetermined-areainformation and the image of the predetermined area T is describedaccording to the embodiment. FIG. 8 is a view illustrating a relationbetween the predetermined-area information and the image of thepredetermined area T. As illustrated in FIG. 8, “ea” denotes anelevation angle, “aa” denotes an azimuth angle, and “α” denotes an angleof view, respectively, of the virtual camera IC. The position of thevirtual camera IC is adjusted, such that the point of gaze of thevirtual camera IC, indicated by the imaging direction (ea, aa), matchesthe central point CP of the predetermined area T as the imaging area ofthe virtual camera IC. The predetermined-area image Q is an image of thepredetermined area T, in the spherical image CE. “f” denotes a distancefrom the virtual camera IC to the central point CP of the predeterminedarea T. L is a distance between the center point CP and a given vertexof the predetermined area T (2L is a diagonal line). In FIG. 8, atrigonometric function equation generally expressed by the followingequation 1 is satisfied.L/f=tan(α/2)  (Equation 1)

<Hardware Configuration of Terminal Data Processing Device>

FIG. 9 is a schematic diagram illustrating a hardware configuration ofthe terminal data processing device 5, according to the embodiment. Asillustrated in FIG. 9, the terminal data processing device 5 includes aCPU 501, a ROM 502, a RAM 503, an EEPROM 504, a CMOS sensor 505, anacceleration and orientation sensor 506, a medium I/F 508, and a GPSreceiver 509.

The CPU 501 controls entire operation of the terminal data processingdevice 5. The ROM 502 stores a control program for controlling the CPU501. The RAM 503 is used as a work area for the CPU 501. The EEPROM 504reads or writes various data such as a control program for the terminaldata processing device under control of the CPU 501. The CMOS sensor 505captures an object (for example, a self-image of the user operating theterminal data processing device 5) under control of the CPU 501 toobtain captured image data. The acceleration and orientation sensor 506includes various sensors such as an electromagnetic compass orgyrocompass for detecting geomagnetism, and an acceleration sensor. Themedium I/F 508 controls reading or writing of data with respect to arecording medium 507 such as a flash memory. The GPS receiver 509receives a GPS signal from a GPS satellite.

The terminal data processing device 5 further includes a long-rangecommunication circuit 511, an antenna 511 a for the long-rangecommunication circuit 511, a camera 512, an imaging element I/F 513, amicrophone 514, a speaker 515, an audio input/output I/F 516, a display517, an external device connection I/F 518, a short-range communicationcircuit 519, an antenna 519 a for the short-range communication circuit519, and a touch panel 521.

The long-range communication circuit 511 is a circuit that communicateswith another device through the intranet 200, for example. The camera112 is an example of built-in imaging device capable of capturing atarget under control of the CPU 501. The imaging element I/F 513 is acircuit that controls driving of the camera 512. The microphone 514 isan example of audio collecting device, which is a built-in type, capableof inputting audio under control of the CPU 401. The audio I/O I/F 516is a circuit for inputting or outputting an audio signal to themicrophone 514 or from the speaker 515 under control of the CPU 501.

The display 517 may be a liquid crystal or organic electro luminescence(EL) display that displays an image of a subject, an operation icon, orthe like. The external device connection I/F 518 is an interface circuitthat connects the terminal data processing device 5 to various externaldevices. The short-range communication circuit 219 is a communicationcircuit that communicates in compliance with the near field radiocommunication (NFC) (Registered Trademark), the Bluetooth (RegisteredTrademark), and the like. The touch panel 521 is an example of inputdevice that enables the user to input a user instruction to the terminaldata processing device 5 through touching a screen of the display 517.

The terminal data processing device 5 further includes a bus line 510.The bus line 510 may be an address bus or a data bus, which electricallyconnects various elements such as the CPU 501 of FIG. 9.

<Hardware Configuration of Video Conference Terminal>

FIG. 10 is a schematic block diagram illustrating a hardwareconfiguration of the video conference terminal 7, according to theembodiment. As illustrated in FIG. 10, the video conference terminal 7includes a CPU 701, a ROM 702, a RAM 703, a flash memory 704, a SSD 705,a medium I/F 707, an operation key 708, a power switch 709, a bus line710, a network I/F 711, a camera 712, an imaging element I/F 713, amicrophone 714, a speaker 115, an audio input/output I/F 716, a displayI/F 717, an external device connection I/F 718, a short-rangecommunication circuit 719, and an antenna 719 a for the short-rangecommunication circuit 719.

The CPU 701 controls entire operation of the video conference terminal7. The ROM 702 stores a control program for controlling the CPU 701 suchas an IPL. The RAM 703 is used as a work area for the CPU 701. The flashmemory 704 stores various data such as a communication control program,image data, and audio data. The SSD 705 controls reading or writing ofvarious data with respect to the flash memory 704 under control of theCPU 701. In alternative to the SSD, a hard disk drive (HDD) may be used.The medium I/F 707 controls reading or writing of data with respect to arecording medium 706 such as a flash memory. The operation key (keys)708 is operated by a user to input a user instruction such as a userselection of a communication destination of the video conferenceterminal 7. The power switch 709 is a switch that receives aninstruction to turn on or off the power of the video conference terminal7.

The network I/F 711 is an interface that controls communication of datawith an external device through the communication network 600. Thecamera 712 is an example of built-in imaging device capable of capturinga target under control of the CPU 701. The imaging element I/F 713 is acircuit that controls driving of the camera 712. The microphone 714 isan example of audio collecting device, which is a built-in type, capableof inputting audio under control of the CPU 701. The audio I/O I/F 716is a circuit for inputting or outputting an audio signal between themicrophone 714 and the speaker 715 under control of the CPU 701. Thedisplay I/F 717 is a circuit for transmitting display data to anexternal display 720 under control of the CPU 701. The external deviceconnection I/F 718 is an interface circuit that connects the videoconference terminal 7 to various external devices. The short-rangecommunication circuit 719 is a communication circuit that communicatesin compliance with the near field radio communication (NFC) (RegisteredTrademark), the Bluetooth (Registered Trademark), and the like.

The bus line 710 may be an address bus or a data bus, which electricallyconnects various elements such as the CPU 701 of FIG. 10.

The display 720 may be a liquid crystal or organic electroluminescence(EL) display that displays an image of a subject, an operation icon, orthe like. The display 720 is connected to the display I/F 717 by a cable732. The cable 732 may be an analog red green blue (RGB) (video graphicarray (VGA)) signal cable, a component video cable, a high-definitionmultimedia interface (HDMI) signal cable, or a digital video interactive(DVI) signal cable. Here, it is assumed that the cable 732 is the HDMIcable. In this disclosure, the display 720 is a generic term fordisplays 720 a and 720 b to be described later. The HDMI cable 732 is ageneric term for the HDMI cables 732 a and 732 b to be described later.

The camera 712 includes a lens and a solid-state imaging element thatconverts an image (video) of a subject to electronic data throughphotoelectric conversion. As the solid-state imaging element, forexample, a complementary metal-oxide-semiconductor (CMOS) or acharge-coupled device (CCD) is used.

The external device connection I/F 718 is capable of connecting anexternal device such as an external camera, an external microphone, oran external speaker through a USB cable or the like. In the case wherean external camera is connected, the external camera is driven inpreference to the built-in camera 712 under control of the CPU 701.Similarly, in the case where an external microphone is connected or anexternal speaker is connected, the external microphone or the externalspeaker is driven in preference to the built-in microphone 714 or thebuilt-in speaker 715 under control of the CPU 701.

The recording medium 706 is removable from the video conference terminal7. The recording medium 706 can be any non-volatile memory that reads orwrites data under control of the CPU 301, such that any memory such asan EEPROM may be used instead of the flash memory 704.

<Hardware Configuration of Electronic Whiteboard>

FIG. 11 is a schematic block diagram illustrating a hardwareconfiguration of the electronic whiteboard 8, according to theembodiment. As illustrated in FIG. 11, the electronic whiteboard 8includes a central processing unit (CPU) 801, a read only memory (ROM)802, a random access memory (RAM) 803, a solid state drive (SSD) 804, anetwork interface (I/F) 805, and an external device connection interface(I/F) 806.

The CPU 801 controls entire operation of the electronic whiteboard 8.The ROM 802 stores a control program for controlling the CPU 801 such asan IPL. The RAM 803 is used as a work area for the CPU 801. The SSD 804stores various data such as the control program for the electronicwhiteboard 8. The network I/F 805 controls communication with anexternal device through the communication network 600. The externaldevice connection I/F 806 controls communication with an externalresource such as a PC 8100, a USB memory 8600, a microphone 8200, aspeaker 8300, and a camera 8400.

The electronic whiteboard 8 further includes a capturing device 811, agraphics processing unit (GPU) 812, a display controller 813, a contactsensor 814, a sensor controller 815, an electronic pen controller 816, ashort-range communication circuit 819, an antenna 819 a for theshort-range communication circuit 819, a power switch 822, andselectable switches 823.

The capturing device 811 captures an image being displayed at theexternal PC 8100 to display the captured image on the display 820. TheGPU 812 is a semiconductor chip dedicated to processing a graphicalimage. The display controller 813 controls display of an image processedat the GPU 812 for output through the display 820 provided with theelectronic whiteboard 8. The contact sensor 814 detects a touch onto thedisplay 820 with a electronic pen 8500 or a user's hand H. The sensorcontroller 815 controls operation of the contact sensor 814. The contactsensor 814 senses a touch input to a specific coordinate on the display820 using the infrared blocking system.

More specifically, the display 820 is provided with two light receivingelements disposed on both upper side ends of the display 820, and areflector frame surrounding the sides of the display 820. The lightreceiving elements emit a plurality of infrared rays in parallel to asurface of the display 820. The light receiving elements receive lightspassing in the direction that is the same as an optical path of theemitted infrared rays, which are reflected by the reflector frame. Thecontact sensor 814 outputs an identifier (ID) of the infrared ray thatis blocked by an object (such as the user's hand) after being emittedfrom the light receiving elements, to the sensor controller 815. Basedon the ID of the infrared ray, the sensor controller 815 detects aspecific coordinate that is touched by the object. The electronic pencontroller 816 communicates with the electronic pen 8500 to detect atouch by the tip or bottom of the electronic pen 8500 to the display820.

The short-range communication circuit 819 is a communication circuitthat communicates in compliance with the NFC, the Bluetooth, and thelike. The power switch 822 turns on or off the power of the electronicwhiteboard 8. The selectable switches 823 are a group of switches foradjusting, for example, brightness and color of a screen displayed onthe display 820.

The electronic whiteboard 8 further includes a bus line 810. The busline 810 may be an address bus or a data bus, which electricallyconnects various elements such as the CPU 801 of FIG. 11.

The contact sensor 814 is not limited to the infrared blocking systemtype, and may be a different type of detector, such as a capacitancetouch panel that identifies the contact position by detecting a changein capacitance, a resistance film touch panel that identifies thecontact position by detecting a change in voltage of two opposedresistance films, or an electromagnetic induction touch panel thatidentifies the contact position by detecting electromagnetic inductioncaused by contact of an object to a display. In addition or inalternative to detecting a touch by the tip or bottom of the electronicpen 8500, the electronic pen controller 816 may also detect a touch byanother part of the electronic pen 8500, such as a part held by a handof the user.

<Hardware Configuration of PC>

FIG. 12 is a schematic block diagram illustrating a hardwareconfiguration of the PC 6, according to the embodiment. Referring toFIG. 12, the PC 6 includes a CPU 601, a ROM 602, a RAM 603, a HD 604, aHDD 605, a medium I/F 607, a display 608, a touch panel 609, a networkI/F 610, a keyboard 611, a mouse 612, a Digital Versatile DiskRewritable (DVD-RW) drive 614, and a bus line 620.

The CPU 601 controls entire operation of the PC 6. The ROM 602 stores acontrol program for controlling the CPU 601. The RAM 603 is used as awork area for the CPU 601. The HD 604 stores various data such as acontrol program. The HDD 605 controls reading or writing of various datato or from the HD 604 under control of the CPU 601. The medium I/F 607controls reading or writing of data with respect to a recording medium606 such as a flash memory.

The display 608 displays various information such as a cursor, menu,window, characters, or image. The touch panel 609 is an example of inputdevice that enables the user to input a user instruction to the PC 6through touching a screen of the display 608. The network I/F 610 is aninterface that controls communication of data with an external devicethrough the communication network 600. The keyboard 611 is one exampleof input device provided with a plurality of keys for allowing a user toinput characters, numerals, or various instructions. The mouse 612 isone example of input device for allowing the user to select a specificinstruction or execution, select a target for processing, or move acurser being displayed. The DVD-RW drive 614 reads or writes variousdata with respect to a DVD ReWritable (DVD-RW) 613, which is one exampleof removable recording medium. In alternative to the DVD-RW, anyrecording medium may be used such as a DVD-R, Blu-ray Disc (Blu-rayDisc), etc.

The PC 6 further includes a bus line 620. The bus line 620 may be anaddress bus or a data bus, which electrically connects various elementssuch as the CPU 601 of FIG. 12.

<Hardware Configuration of Communication Management Server>

FIG. 12 is a schematic block diagram illustrating a hardwareconfiguration of the communication management server 9, according to theembodiment. Referring to FIG. 12, the communication management server 9,which is implemented by the general-purpose computer, includes a CPU901, a ROM 902, a RAM 903, a HD 904, a HDD 905, a medium I/F 907, adisplay 908, a touch panel 909, a network I/F 910, a keyboard 911, amouse 912, a CD-RW drive 914, and a bus line 920. Since thecommunication management server 9 operates as a server, an input devicesuch as the keyboard 911 and the mouse 912, or an output device such asthe display 908 does not have to be provided.

The CPU 901 controls entire operation of the communication managementserver 9. The ROM 902 stores a control program for operating the CPU901. The RAM 903 is used as a work area for the CPU 901. The HD 904stores various data such as a control program. The HDD 905 controlsreading or writing of various data to or from the HD 904 under controlof the CPU 901. The medium I/F 907 controls reading or writing of datawith respect to a recording medium 906 such as a flash memory.

The display 908 displays various information such as a cursor, menu,window, characters, or image. The touch panel 909 is an example of inputdevice that enables the user to input a user instruction to thecommunication management server 9 through touching a screen of thedisplay 908. The network I/F 910 is an interface that controlscommunication of data with an external device through the communicationnetwork 600. The keyboard 911 is one example of input device providedwith a plurality of keys for allowing a user to input characters,numerals, or various instructions. The mouse 912 is one example of inputdevice for allowing the user to select a specific instruction orexecution, select a target for processing, or move a curser beingdisplayed. The CD-RW drive 914 reads or writes various data with respectto a Compact Disc ReWritable (CD-RW) 913, which is one example ofremovable recording medium.

The communication management server 9 further includes a bus line 920.The bus line 920 is an address bus or a data bus, which electricallyconnects the elements in FIG. 12 such as the CPU 901.

<Hardware Configuration of Relay Device>

The relay device 10 basically has the same configuration as thecommunication management server 9.

Referring to FIG. 12, the relay device 10, which is implemented by thegeneral-purpose computer, includes a CPU 1001, a ROM 1002, a RAM 1003, aHD 1004, a HDD 1005, a medium I/F 1007, a display 1008, a touch panel1009, a network I/F 1010, a keyboard 1011, a mouse 1012, a CD-RW drive1014, and a bus line 1020.

Since the relay device 10 operates as a router, an input device such asthe keyboard 1011 and the mouse 1012, or an output device such as thedisplay 1008 does not have to be provided.

Since the CPU 1001, ROM 1002, RAM 1003, HD 1004, HDD 1005, recordingmedium 1006, medium I/F 1007, display 1008, touch panel 1009, networkI/F 1010, keyboard 1011, mouse 1012, CD-RW drive 1014, and bus line1020, are substantially similar in structure to the CPU 901, ROM 902,RAM 903, HD 904, HDD 905, recording medium 906, medium I/F 907, display908, touch panel 909, network I/F 910, keyboard 911, mouse 912, CD-RWdrive 914, and bus line 920, respectively, description thereof isomitted.

[Functional Configuration of Communication System]

Referring now to the drawings, a functional configuration of thecommunication system 1 is described according to the embodiment. First,referring to FIG. 13 and FIG. 14, functional configurations ofterminals, apparatuses, and servers in the communication system 1 aredescribed, according to the embodiment. FIG. 13 is a diagramillustrating a functional configuration of the communication system 1.

<Functional Configuration of Real-Time Data Processing Terminal>

Referring to FIG. 13, the real-time data processing terminal 3 includesa determiner 33, an image processing unit 34, a connection unit 38, astoring and reading unit 39, and a communication unit 48. These unitsare functions that are implemented by or that are caused to function byoperating any of the hardware components illustrated in FIG. 2 incooperation with the instructions of the CPU 301 according to thecontrol program expanded from the EEPROM 304 to the RAM 303.

The real-time data processing terminal 3 further includes a storage unit3000, which is implemented by the ROM 302, the RAM 303, and/or theEEPROM 304 illustrated in FIG. 2.

Referring to FIG. 13, a functional configuration of the real-time dataprocessing terminal 3 is described according to the embodiment. Thedeterminer 33, which is implemented by instructions of the CPU 301,performs various determinations. The image processing unit 34, which isimplemented by instructions of the CPU 301, digitizes an image inputfrom the imaging unit 40 to generate at least one of video image dataand still image data. For example, the image processing unit 34generates video image data in real time (for example, 1/60 secondinterval) or generates still image data. The connection unit 38, whichis implemented by the imaging unit I/F 313 and the instructions of theCPU 301, is an interface for mechanically and electrically connectingthe imaging unit 40 to the real-time data processing terminal 3. Thestoring and reading unit 39, which is implemented by instructions of theCPU 301, stores various data or information in the storage unit 3000 orreads out various data or information from the storage unit 3000. Thecommunication unit 48, which is implemented by the external deviceconnection I/F 318 and the instructions of the CPU 301, transmits orreceives various data (or information) to or from the communication unit58 of the terminal data processing device 5. The one-to-onecommunication between the communication unit 48 and the communicationunit 58 may be performed via a wired network or a wireless network.

<Functional Configuration of Terminal Data Processing Device>

As illustrated in FIG. 13, the terminal data processing device 5includes a transmitter and receiver 51, a determiner 53, a communicationunit 58, and a storing and reading unit 59. These units are functionsthat are implemented by or that are caused to function by operating anyof the elements illustrated in FIG. 9 in cooperation with theinstructions of the CPU 501 according to the control program expandedfrom the EEPROM 504 to the RAM 503. The terminal data processing device5 further includes a storage unit 5000, which is implemented by the ROM502, RAM 503 and EEPROM 504 illustrated in FIG. 9.

Referring to FIG. 13, a functional configuration of the terminal dataprocessing device 5 is described according to the embodiment. Thetransmitter and receiver 51 of the terminal data processing device 5,which is implemented by the long-range communication circuit 511, theantenna 511 a, and the instructions of the CPU 501, transmits orreceives various data (or information) to or from another terminal,device, or server via the communication network 600. The determiner 53,which is implemented by instructions of the CPU 501, performs variousdeterminations. The communication unit 58, which is implemented by theexternal device connection I/F 518, and the instructions of the CPU 501,transmits or receives various data (or information) to or from thecommunication unit 48 of the real-time data processing terminal 3. Theone-to-one communication between the communication unit 58 and thecommunication unit 48 may be performed via a wired network or a wirelessnetwork. The storing and reading unit 59, which is implemented byinstructions of the CPU 501, stores various data or information in thestorage unit 5000 or reads out various data or information from thestorage unit 5000.

<Functional Configuration of PC>

As illustrated in FIG. 13, the PC 6 includes a transmitter and receiver61, an acceptance unit 62, a spherical image and audio processor 64, adisplay control 67, and a storing and reading unit 69.

These units are functions that are implemented by or that are caused tofunction by operating any of the hardware components illustrated in FIG.11 in cooperation with the instructions of the CPU 601 according to thecontrol program expanded from the HD 604 to the RAM 603. The PC 6further includes a storage unit 6000, which is implemented by the ROM602, RAM 603, or HD 604 as illustrated in FIG. 11.

The transmitter and receiver 61 of the PC 6, which is implemented by thenetwork I/F 610 and the instructions of the CPU 601, transmits orreceives various data (or information) to or from another terminal,apparatus, or server via the communication network 600.

The acceptance unit 62 is implemented by the keyboard 611 or mouse 612,which operates under control of the CPU 601, to receive variousselections or inputs from the user. The spherical image and audioprocessor 64 processes spherical video image data received from therelay device 10 to display a spherical video image on the display 608,and processes audio data received from the relay device 10 to outputaudio through an external speaker.

The spherical image and audio processor 64 is capable of reproducingspherical still image data, planar video image data, and planar stillimage data, in addition to spherical video image data.

The display control 67, which is implemented by the instructions of theCPU 601 of the PC 6, controls the display 608 to display various typesof image, such as a spherical video image, spherical still image, planarvideo image, or planar still image.

The storing and reading unit 69, which is implemented by instructions ofthe CPU 601 of the PC 6, stores various data or information in thestorage unit 6000 or reads out various data or information from thestorage unit 6000.

<Functional Configuration of Video Conference Terminal>

As illustrated in FIG. 13, the video conference terminal 7 includes atransmitter and receiver 71, an acceptance unit 72, a planar video imageand audio processor 74, a display control 77, an audio control 78, and astoring and reading unit 79.

These units are functions that are implemented by or that are caused tofunction by operating any of the elements of the video conferenceterminal 7 illustrated in FIG. 10 in cooperation with the instructionsof the CPU 701 according to the control program expanded from the flashmemory 704 to the RAM 703. The video conference terminal 7 furtherincludes a storage unit 7000, which is implemented by the ROM 702, RAM703, or flash memory 704 of FIG. 10.

The transmitter and receiver 71 of the video conference terminal 7,which is implemented by the network I/F 711 and the instructions of theCPU 701, transmits or receives various data (or information) to or fromother server, apparatus, terminal, or device via a communication network600. The acceptance unit 72, which is implemented by the operation key708, operating under control of the CPU 701, receives various selectionsor inputs from the user.

The planar video image and audio processor 74 processes planar videoimage data received from the relay device 10 to display video at thedisplay 720, and processes audio data received from the relay device 10to output audio through the speaker 715. The planar video image andaudio processor 74 is not capable of reproducing spherical video imagedata, spherical still image data, and planar still image data.

The display control 77, which is implemented by the instructions of theCPU 701 of the video conference terminal 7, controls the display 720 todisplay various types of image, such as the planar video image. Theaudio control 78, which is implemented by the instructions of the CPU701 of the video conference terminal 7, controls collection of audio atthe microphone 714 or output of audio through the speaker 715. Thestoring and reading unit 79, which is implemented by instructions of theCPU 701 of the video conference terminal 7, stores various data orinformation in the storage unit 7000 or reads out various data orinformation from the storage unit 7000.

<Functional Configuration of Electronic Whiteboard>

As illustrated in FIG. 13, the electronic whiteboard 8 includes atransmitter and receiver 81, an acceptance unit 82, a planar still imageprocessor 84, a display control 87, and a storing and reading unit 89.

These units are functions that are implemented by or that are caused tofunction by operating any of the hardware components of the electronicwhiteboard 8 illustrated in FIG. 11 in cooperation with the instructionsof the CPU 801 according to the control program expanded from the SSD804 to the RAM 803. The electronic whiteboard 8 further includes astorage unit 8000, which is implemented by the ROM 802, RAM 803, or SSD804 of FIG. 11.

The transmitter and receiver 81 of the electronic whiteboard 8, which isimplemented by the I/F 805 and the instructions of the CPU 801,transmits or receives various data (or information) to or from anotherterminal, apparatus, or server via the communication network 600. Theacceptance unit 82, which is implement by the selectable switches 823 ofthe electronic whiteboard 8, operating under control of the CPU 801,receives various selections or inputs from the user.

The planar still image processor 84 processes the planar still imagedata transmitted from the relay device 10 to display a planar stillimage on the display 820. The planar still image processor 84 is notcapable of reproducing spherical video image data, spherical still imagedata, and planar video image data. In a substantially similar manner forthe planar video image and audio processor 74, the planar still imageprocessor 84 may be provided with capability of reproducing audio datareceived from the relay device 10 to output audio through the speaker8300.

The display control 87, which is implemented by the instructions of theCPU 801 of the electronic whiteboard 8, controls the display 820 todisplay various types of image, such as a planar still image.

The storing and reading unit 89, which is implemented by instructions ofthe CPU 801 of the electronic whiteboard 8, stores various data orinformation in the storage unit 8000 or reads out various data orinformation from the storage unit 8000.

<Functional Configuration of Communication Management Server 9>

As illustrated in FIG. 13, the communication management server 9includes a transmitter and receiver 91, a determiner 93, and a storingand reading unit 99. These units are functions that are implemented byor that are caused to function by operating any of the hardwarecomponents illustrated in FIG. 12 in cooperation with the instructionsof the CPU 901 according to the control program expanded from the HD 904to the RAM 903. The communication management server 9 further includes astorage unit 9000, which is implemented by the ROM 902, the RAM 903 andthe HD 904 illustrated in FIG. 12.

The storage unit 9000 stores therein a communication management DB(database) 9001, a transmittable image type management DB 9002, and areproducible image type management DB 9003. The communication managementDB 9001 is implemented by a communication management table of FIG. 14A.The transmittable image type management DB 9002 is implemented by atransmittable image type management table of FIG. 14B. The reproducibleimage type management DB 9003 is implemented by a reproducible imagetype management table of FIG. 14C.

FIG. 14A is a conceptual diagram illustrating an example communicationmanagement table. The communication management table includes a deviceID for identifying a transmission source (“sender”), and a device ID foridentifying a transmission destination (“receiver”), in association. Asdescribed below, in one example, the communication management tablestores, for each session that has been established, a device ID of asender that transmits image data, and a device of a receiver thatreceives image data.

FIG. 14B is a conceptual diagram illustrating an example transmittableimage type management table. The transmittable image management tablestores a device ID for identifying a transmission source, and a type ofimage that is transmittable by the transmission source. “None” in thetable represents a state in which no value is set. “V3” represents athree-dimensional, spherical video image. “V2” represents atwo-dimensional, planar video image. “S” represents a two-dimensional,planar still image. When the transmission source is in an initial state,and has not yet transmitted image data, no information regarding a typeof transmittable image is stored in the table, such that the value isset to “None”. For example, the initial state may be a state in whichthe transmission source in the communication system 1 has justestablished a communication session with a transmission destination, andno data has been exchanged through the communication session.

FIG. 14C is a conceptual diagram illustrating an example reproducibleimage type management table. The reproducible image type managementtable includes a device ID for identifying a transmission destination(“receiver”), and a type of image that is reproducible by thetransmission destination. Information on the type of image reproducibleby each device as a receiver is previously stored, for example, for eachapparatus that can be a receiver according to a user operation.

Information on a type of image that is transmittable by the transmissionsource, which is stored in the transmittable image type management tableof FIG. 14B, is dynamically changed, according to a request from thetransmission destination (that is, the receiver), as described belowreferring to FIG. 15.

The table in FIG. 14B illustrates the case where the type oftransmittable image for the image acquisition terminal 2 identified withthe device ID “01aa” is changed from “None” to “S”, and further changedfrom “S” to “S, V2”. In this disclosure, the device ID of thetransmission source (sender) is an example of sender identifier.

Next, referring to FIG. 13, a functional configuration of thecommunication management server 9 is described in detail. Thetransmitter and receiver 91 of the communication management server 9,which is implemented by the network I/F 711 and the instructions of theCPU 701, transmits or receives various data (or information) to or fromother server, apparatus, terminal, or device via a communication network600. The determiner 93, which is implemented by instructions of the CPU901, performs various determinations. The storing and reading unit 99,which is implemented by instructions of the CPU 901, stores various dataor information in the storage unit 9000 or reads out various data orinformation from the storage unit 9000.

<Functional Configuration of Relay Device>

As illustrated in FIG. 13, the relay device 10 includes a transmitterand receiver (transferrer) 101, a determiner 103, an image processingunit 104, and a storing and reading unit 109.

These units are functions that are implemented by or that are caused tofunction by operating any of the hardware components illustrated in FIG.12 in cooperation with the instructions of the CPU 901 according to thecontrol program expanded from the HD 904 to the RAM 903. The relaydevice 30 further includes a storage unit 10000 implemented by the ROM902, RAM 903, and HD 904 illustrated in FIG. 12. The storage unit 10000includes a transfer image type management DB 10001. The transfer imagetype management DB 10001 is implemented by a transfer image typemanagement table, which is described below.

FIG. 14D is a conceptual diagram illustrating an example transfer imagetype management table. The transfer image type management table includesa device ID for identifying a transmission source (sender), a device IDfor identifying a transmission destination (receiver) capable ofreproducing spherical video image data, a device ID of a receivercapable of reproducing planar video image data, and a device ID of areceiver capable of reproducing planar still image data, in association.The device ID of the transmission destination (receiver) is an exampleof receiver identifier.

Referring to FIG. 13, a functional configuration of the relay device 10is described in detail. The transmitter and receiver 101 of the relaydevice 10, which is implemented by the network I/F 1010 and theinstructions of the CPU 1001, transmits or receives various data (orinformation) to or from other server, apparatus, terminal, or device viaa communication network 600.

The determiner 103, which is implemented by instructions of the CPU1001, performs various determinations. The image processing unit 104performs image processing on spherical video image data, planar videoimage data, and planar still image data. The image processing unit 104may perform image processing on spherical still image data.

The storing and reading unit 109, which is implemented by instructionsof the CPU 1001, stores various data or information in the storage unit10000 or reads out various data or information from the storage unit10000.

[Operation]

Referring to FIGS. 15 to 18, operation of managing image types to beprocessed, performed by the communication system 1, is describedaccording to the embodiment. FIG. 15 is a data sequence diagramillustrating example operation of transmitting and receiving image data,between the image acquisition terminal 2 as a sender, and the electronicwhiteboard 8 and the video conference terminal 7 each being a receiver.

Accordingly, when a communication session is established, thecommunication management server 9 creates a new entry in thecommunication management table of FIG. 14A, which includes the device ID“01aa” of the image acquisition terminal 2 in the “device ID of sender”field, and the device ID “02aa” of the electronic whiteboard 8 and thedevice ID “02ab” of the video conference terminal 7 in the “device ID ofreceiver” field.

The following assumes that the imaging unit 40 b in FIG. 3 is connectedto the real-time data processing terminal 3, and the real-time dataprocessing terminal 3 acquires the spherical video image data from theimaging unit 40 b. For the descriptive purposes, description onacquisition of audio data is omitted. While the image acquisitionterminal 2 is capable of transmitting spherical video image data, one ormore devices at the receiver side may not be able to process sphericalvideo image data.

More specifically, as indicated by the reproducible image typemanagement table of FIG. 14C, the electronic whiteboard 8 with thedevice ID “02aa” is capable of reproducing planar still image data(“S”), but not capable of reproducing spherical video and still imagedata, and planar still video data. The video conference terminal 7 withthe device ID “02ab” is capable of reproducing planar video image data(“V2”), but not capable of reproducing planar still image data andspherical video and still image data.

First, the case where the electronic whiteboard 8 sends a request fortransmission of image data is described.

As illustrated in FIG. 15, at the electronic whiteboard 8, as the powerswitch 822 is turned on, the transmitter and receiver 81 of theelectronic whiteboard 8 transmits a notification indicating that theelectronic whiteboard 8 is ready to receive image data, to thecommunication management server 9 (S11). This notification includes adevice ID of a transmission source (sender) for identifying a requesttransmission target, and a device ID of a request transmission source(receiver). The request transmission source, which is the receiver, isany device that has sent the notification indicating that image data canbe received, when starting communication with the request transmissiontarget (sender). In this specific example, the electronic whiteboard 8is the receiver, and the image acquisition terminal 2 is sender. Thetransmitter and receiver 91 of the communication management server 9receives the notification indicating that image data can be received.

The transmitter and receiver 91 of the communication management server 9transmits a transmission start request for requesting to starttransmission of image data, to the terminal data processing device 5,via the communication network 600 (S12). The transmitter and receiver 51of the terminal data processing device 5 receives the transmission startrequest.

Next, the determiner 53 of the terminal data processing device 5determines whether to request transmission of image data (S13).

Referring to FIG. 16, S13 of determining whether to request transmissionof image data is described as an example. FIG. 16 is a flowchartillustrating processing to determine whether to request transmission ofimage data according to an example. As illustrated in FIG. 16, thedeterminer 53 determines whether transmission of image data has beenstarted (S13-1). When transmission of image data has been started (“YES”at S13-1), S13 ends. In such case, the operation does not proceed toS17. When transmission of image data has not started (“NO” at S13-1),operation proceeds to S17 to transmit a transmission start request tothe terminal data processing device 5. With this transmission startrequest, the image acquisition terminal 2 starts transmission of imagedata, in case transmission of image data has not been started.

This processing of S13 is optional, as the image acquisition terminal 2is programmed to constantly transmit the image data to the relay device10.

In response to receiving the notification at S11, the communicationmanagement server 9 determines a type of image data that can be receivedat the transmission destination (S14).

Referring now to FIG. 17, processing to determine a type of image isdescribed according to an example. FIG. 17 is a flowchart illustratingprocessing to determine an image type in this example.

As illustrated in FIG. 17, the storing and reading unit 99 stores, inthe communication management DB 9001 (FIG. 14A), the device ID of atransmission destination, included in the notification indicating thatimage data can be received (S14-1). In this case, a new entry isgenerated in the communication management DB 9001 for a communicationestablished between the image acquisition terminal 2 and the electronicwhiteboard 8. Accordingly, the device ID “01aa” of the image acquisitionterminal 2 as the sender, and the device ID “02aa” of the electronicwhiteboard 8 as the receiver are stored in association.

The storing and reading unit 99 searches the transmittable image typemanagement DB 9002 (FIG. 14B) using the sender device ID received atS11, as a search key, to obtain a type of image that is transmittable bythe transmission source as the sender (S14-2). In this example, theimage acquisition terminal 2 is a sender.

The storing and reading unit 99 further searches the reproducible imagetype management DB 9003 (FIG. 14C) using the receiver device ID receivedat S11, as a search key, to obtain a type of image that is reproducibleby the transmission destination as the receiver (S14-3). In thisexample, the electronic whiteboard 8 is a receiver. S14-2 and S14-3 maybe performed in any order.

Next, the determiner 93 determines whether there is any image type thathas been registered to the transmittable image type management DB 9002,for example, as a result of S14-2 (S14-4).

When there is no image type registered (“NO” at S14-4), the storing andreading unit 99 refers to the transmittable image type management DB9002, to change a transmittable image type of the sender with the senderdevice ID received at S11, from “None” to the image type that isobtained at S14-3 for the receiver with the receiver ID received at S11(S14-5). In this specific example, since the electronic whiteboard 8 hastransmitted the notification, as the receiver, the image type in thetransmittable image type management table of FIG. 14B is changed from“None” to “S” indicating a type of image reproducible by the electronicwhiteboard 8 that is obtained at S14-3. The operation then ends.

Next, the case where the video conference terminal 7 sends a request fortransmission of image data is described. For the descriptive purposes,it is assumed that the request for transmission received from the videoconference terminal 7 is processed after processing the request fortransmission received from the electronic whiteboard 8. However, thiscan be processed before, or at substantially the same time with, thetime when the request form the electronic whiteboard 8 is processed.

S14-1 to S14-4 are performed in a substantially similar manner asdescribed above. Specifically, in response to the notification receivedat S11, the communication management server 9 adds the device ID “02ab”of the video conference terminal 7 in the “device ID of the receiver”field, in association with the device ID “01aa” of the image acquisitionterminal 2 as the sender.

At S14-4, it is determined that there is an image type that has beenregistered to the transmittable image type management DB 9002. When thedeterminer 93 determines that information on the image type has beenregistered at S14-4 (“YES” at S14-4), operation proceeds to S14-6. AtS14-6, the determiner 93 determines whether the transmittable image typeobtained at S14-2, and the reproducible image type obtained at S14-3match (S14-6).

When it is determined that the image types match (“YES” at S14-6), theoperation ends to proceed to S15 of FIG. 15. When the determiner 93determines that the image types differ (“NO” at S14-6), the storing andreading unit 99 updates an entry of the transmittable image typemanagement DB 9002, to include the image type stored in the reproducibleimage type management DB 9003, obtained at S14-3, in association withthe device ID of the receiver (S14-5).

In this specific example, the reproducible image type of the receiverthat has obtained at S14-3 is “V2”, which differs from “S” stored in thetransmittable image type management table of FIG. 14B for the sender ID“01aa”. In such case, the image type “S” is changed to “S, V2”, suchthat the reproducible image type “V2” for the video conference terminal7 (“02ab”) is added. The operation then ends to proceed to S15 of FIG.15.

Through operation of FIG. 17, information on types of image to betransmitted by the sender can be dynamically changed.

Referring back to FIG. 15, after S14, the transmitter and receiver 91 ofthe communication management server 9 transmits a transfer request forrequesting transfer of image data, to the relay device 10 (S15). Thetransfer request includes the device ID of the sender and the device IDof the receiver, received at S11, and information on the image typeobtained at S14. Specifically, the information on the image type isobtained from the transmittable image type management table of FIG. 14B.That is, the information on the image type in the transmittable imagetype management table of FIG. 14B indicates the type of imagetransmittable by the sender and reproducible by the receiver. Thetransmitter and receiver 101 of the relay device 10 receives thetransfer request.

Next, at the relay device 10, the storing and reading unit 109 storesthe device ID of the sender and the device ID of the receiver, includedin the transfer request, in the transfer image type management DB 10001(S16). In this specific example, the device ID “02aa” of the electronicwhiteboard 8 as the receiver that can reproduce the planar still imagedata “S” is stored in the “device ID of receiver for planar still image”field. The device ID “02ab” of the video conference terminal 7 as thereceiver that can reproduce the planar video image data “V2” is storedin the “device ID of receiver for planar video image” field.

At the real-time data processing terminal 3, the image processing unit34 encodes the spherical video image data (S18). The communication unit48 outputs data of an equirectangular projection image as illustrated inFIG. 4C, to the communication unit 58 of the terminal data processingdevice 5 (S19). The transmitter and receiver 51 of the terminal dataprocessing device 5 transmits the data of the equirectangular projectionimage, received at the communication unit 58, to the relay device 10through the communication network 600 (S20). The transmitter andreceiver 101 of the relay device 10 receives the data of equirectangularprojection image.

The relay device 10 determines a type of image that is transferrable tothe transmission destination (S21). The processing of determining animage type that is transferrable is described in detail with referenceto FIG. 18. FIG. 18 is a flowchart illustrating processing to determinean image type of image to be transferred.

The storing and reading unit 109 of the relay device 10 searches thetransfer image type management DB 10001 using the device ID of sender asa search key to obtain a value in the “device ID of receiver forspherical video image” field (S21-1). The device ID of sender has beenobtained at the relay device 10 from the terminal data processing device5 (image acquisition terminal 2), when a communication session isestablished between the image acquisition terminal 2 and the relaydevice 10 before performing operation of FIG. 15.

The determiner 103 determines whether any receiver is capable ofreproducing spherical video image data, based on whether any device IDis stored in the “device ID of receiver for spherical video image” ofthe transfer image type management DB 10001 (S21-2).

When it is determined that any device ID is stored, that is, there isany receiver capable of reproducing spherical video image data (“YES” atS21-2), the transmitter and receiver 101 transfers spherical video imagedata to the receiver having the device ID stored in the “device ID ofreceiver for spherical video image” field (S22). For example, based onthe data of equirectangular projection image, the relay device 10generates spherical video image data, and transmit the spherical videoimage data to the receiver. The receiver can then receive the sphericalvideo image data.

When it is determined that no device ID is stored, that is, there is noreceiver capable of reproducing spherical video image data (“NO” atS21-2), the transmitter and receiver 101 does not send spherical videoimage data.

In this specific example, since there is no device ID stored in the“device ID of receiver for spherical video image” field, spherical videoimage data is not transmitted at S22.

The storing and reading unit 109 searches the transfer image typemanagement DB 10001 using the device ID of sender as a search key toobtain a value in the “device ID of receiver for planar video image”field (S21-3).

The determiner 103 determines whether any receiver is capable ofreproducing planar video image data, based on whether any device ID isstored in the “device ID of receiver for planar video image” of thetransfer image type management DB 10001 (S21-4).

When it is determined that any device ID is stored, that is, there isany receiver capable of reproducing planar video image data (“YES” atS21-4), the image processing unit 104 selects a part of the sphericalvideo image defined by a first imaging direction and a first angle ofview (called “first partial area”), for each of frames of sphericalvideo image data (S21-5), to generate planar video image data. Thetransmitter and receiver 101 transfers planar video image data, whichincludes the first partial area of spherical video image data, to thereceiver having the device ID stored in the “device ID of receiver forplanar video image” field (S22). The receiver can then receive theplanar video image data indicating the first partial area of sphericalvideo image. The first imaging direction and the first angle of view maybe previously set, for example, according to the user preference.

When it is determined that no device ID is stored, that is, there is noreceiver capable of reproducing planar video image data (“NO” at S21-4),the transmitter and receiver 101 does not send planar video image data.

In this specific example, since there is one device ID “02ab” stored inthe “device ID of receiver for planar video image” field, planar videoimage is transmitted at S22 to the video conference terminal 7.

The storing and reading unit 109 searches the transfer image typemanagement DB 10001 using the device ID of sender as a search key toobtain a value in the “device ID of receiver for planar still image”field (S21-6).

The determiner 103 determines whether any receiver is capable ofreproducing planar still image data, based on whether any device ID isstored in the “device ID of receiver for planar still image” of thetransfer image type management DB 10001 (S21-7).

When it is determined that any device ID is stored, that is, there isany receiver capable of reproducing planar still image data (“YES” atS21-7), the image processing unit 104 selects a part of the sphericalvideo image defined by a second imaging direction and a second angle ofview (called “second partial area”), for each of frames of sphericalvideo image data (S21-8), to generate planar still image data. Thesecond imaging direction and the second angle of view may be previouslyset, for example, according to the user preference.

The image processing unit 104 encodes the planar still image dataincluding the second partial area of spherical video image data (S21-9).The transmitter and receiver 101 transfers the planar still image data,which is encoded, to the receiver having the device ID stored in the“device ID of receiver for planar still image” field (S22).

When it is determined that no device ID is stored, that is, there is noreceiver capable of reproducing planar still image data (“NO” at S21-7),the transmitter and receiver 101 does not send planar still image data.

In this specific example, since there is the device ID “02aa” of theelectronic whiteboard 8 stored in the “device ID of receiver for planarstill image” field, the transmitter and receiver 101 transfers theplanar still image data to the electronic whiteboard 8. The electronicwhiteboard 8 as the receiver receives the planar still image data.

Next, referring to FIG. 15, the receiver reproduces the image data thatis received (S23). In this specific example, the electronic whiteboard8, which receives the planar still image data at S22, processes theplanar still image data to display a planar still image on the display820. The video conference terminal 7, which receives the planar videoimage data at S22, processes the planar video image data to display aplanar video image on the display.

As described above, when a transmittable image type for the sender(image acquisition terminal 2) and a reproducible image type for thereceiver (video conference terminal 7, electronic whiteboard 8) dot notmatch, the relay device 10 applies image processing to image datareceived from the sender, to generate image data that is compatible withthe receiver.

In one example, when a new apparatus is introduced, information on thetransmittable image type for the image acquisition terminal 2 isupdated, to include an image type that is reproducible by the newlyintroduced apparatus. Specifically, as described above referring to FIG.15, the transmittable image type management table of FIG. 14B isupdated, based on information obtained from the reproducible image typemanagement table of FIG. 14C. The updated information is then recordedto the transfer image type management table of FIG. 14D. The relaydevice 10 refers to the updated transfer image type management table todetermine a type of image data to be transferred to each apparatus as areceiver. With this configuration, the apparatus as the receiver is ableto reproduce image data.

While the above-described example illustrated in FIGS. 15 to 18describes the case where the electronic whiteboard 8 and the videoconference terminal 7 are newly introduced, the above-describedoperation of FIGS. 15 to 18 is applicable to other example case, such asthe example case in which the image type for any exiting apparatus ismodified.

The above-described embodiments are illustrative and do not limit thepresent invention. Thus, numerous additional modifications andvariations are possible in light of the above teachings. For example,elements and/or features of different illustrative embodiments may becombined with each other and/or substituted for each other within thescope of the present invention.

For example, any processing to be performed by any one of theprocessors, such as the CPU 301, 501, 701, 901, and 1001 may beperformed by any combination of these processors. In one example, thecommunication management server 9 may be performed by a single processoror a plurality of processors.

Further, any one of data or information stored in databases may becollectively managed with any number of database. For example, thetransmittable image type management DB 9002 (FIG. 14B) and thereproducible image type management DB 9003 (FIG. 14C) may be integratedinto one database.

Moreover, the communication management server 9 and the relay device 10may be integrated into one system. In such case, communication betweenthe relay device 10 and the communication management server 9, such asprocessing of S15, does not have to be performed. Further, processing todetermine a type of image that is transmittable by the transmissionsource and reproducible by the transmission destination may be performedat once at S14. In such case, the transfer image type management DB10001 does not have to be provided, if the same information can bemanaged using the transmittable image type management DB 9002, which canbe modified based on the reproducible image type management DB 9003.

Further, the above-described operation of FIG. 18 may be performed invarious other ways. For example, the relay device 10 may determinewhether the receiver is capable of reproducing video image data. When itis determined that the receiver is not capable of reproducing videoimage data, the relay device 10 may apply processing to generate stillimage data from the video image data. For example, the relay device 10may select one of a plurality of frames of video image data to generatestill image data. Since still image data can generally processed at anyapparatus, the relay device 10 may simply determine to convert thespherical video image data to planar still image data.

Each of the functions of the described embodiments may be implemented byone or more processing circuits or circuitry. Processing circuitryincludes a programmed processor, as a processor includes circuitry. Aprocessing circuit also includes devices such as an application specificintegrated circuit (ASIC), digital signal processor (DSP), fieldprogrammable gate array (FPGA), System on a chip (SOC), GPU, andconventional circuit components arranged to perform the recitedfunctions.

The invention claimed is:
 1. A communication system, comprising: a relayapparatus, communicatively coupled to a first apparatus and a secondapparatus over a network, for transferring image data transmitted from atransmission source to a transmission destination, the relay apparatusincluding circuitry configured to: receive video image data from thefirst apparatus being the transmission source; determine whether thesecond apparatus being a transmission destination is configured toreproduce the video image data received from the first apparatus; basedon a determination that the second apparatus is not configured toreproduce the video image data received from the first apparatus, applyimage processing to the video image data to generate converted imagedata compatible with the second apparatus; and transfer the convertedimage data compatible with the second apparatus to the second apparatus;and a server, which controls communication between the first apparatusand the second apparatus, the server including: a memory that stores,for each transmission source that has been registered, information on atype of image that is transmittable by the transmission source andreproducible by the transmission destination; and another circuitryconfigured to: receive a request for transmitting image data from thesecond apparatus, the request including an identifier of the firstapparatus being the transmission source and an identifier of the secondapparatus being the transmission destination; update the information onthe type of image stored for the first apparatus to include a type ofimage that is reproducible by the second apparatus, as the type of imagethat is transmittable by the first apparatus and reproducible by thesecond apparatus; and transmit a transfer request for transferring thevideo image data to the relay apparatus, the transfer request includinginformation on the type of image that is transmittable by the firstapparatus and reproducible by the second apparatus, wherein thecircuitry of the relay apparatus is configured to determine whether thesecond apparatus is configured to reproduce the video image data, basedon the information on the type of image that is received from theserver.
 2. The communication system of claim 1, wherein in a case thatthe determination indicates that the second apparatus is not configuredto reproduce the video image data, but configured to reproduce stillimage data, the circuitry is further configured to select one of aplurality of frames of the video image data to generate still imagedata, and transfer the generated still image data to the secondapparatus.
 3. The communication system of claim 1, wherein in a casethat the video image data received from the first apparatus is sphericalvideo image data, and the determination indicates that the secondapparatus is not configured to reproduce the spherical video image data,the circuitry is further configured to: determine whether the secondapparatus is configured to reproduce planar still image data, and in acase that the second apparatus is determined as being configured toreproduce planar still image data, generate planar still image datacorresponding to a specific area of the spherical video image data, thespecific area being defined by a specific imaging direction and aspecific angle of view, and transfer the planar still image data to thesecond apparatus.
 4. The communication system of claim 1, wherein in acase that the video image data received from the first apparatus isspherical video image data, and the determination indicates that thesecond apparatus is not configured to reproduce the spherical videoimage data, the circuitry is further configured to determine whether thesecond apparatus is configured to reproduce planar video image data, andin a case that the circuitry determines that the second apparatus isconfigured to reproduce planar video image data, the circuitry isfurther configured to generate planar video image data corresponding toa specific area of the spherical video image data, the specific areabeing defined by a specific imaging direction and a specific angle ofview, and transfer the planar video image data to the second apparatus.5. The communication system of claim 4, wherein the circuitry is furtherconfigured to: in a case that determining that the second apparatus isnot configured to reproduce planar video image data, generate planarstill image data corresponding to a specific area of the spherical videoimage data, the specific area being defined by a specific imagingdirection and a specific angle of view, and transfer the planar stillimage data to the second apparatus.
 6. The communication system of claim1, wherein the circuitry is further configured to: receive, from aserver, which controls communication between the first apparatus and thesecond apparatus, a transfer request including an identifier of thefirst apparatus, an identifier of the second apparatus, and informationon a type of image that is both transmittable by the first apparatus andreproducible by the second apparatus; and determine whether the secondapparatus is configured to reproduce the video image data received fromthe first apparatus, based on the information on the type of image thatis both transmittable by the first apparatus and reproducible by thesecond apparatus.
 7. The communication system of claim 1, wherein therelay apparatus further includes another memory that stores, for eachparticular transmission source of one or more transmission source,information indicating a type of image that is both transmittable by theparticular transmission source and reproducible by a particulartransmission destination, the information including an identifier of theparticular transmission destination.
 8. The communication system ofclaim 7, wherein the circuitry is further configured to search the othermemory, using an identifier of the first apparatus as a key, for anidentifier of the second apparatus and a corresponding type of imagethat is reproducible by the second apparatus, the type of image beingone of a spherical video image, a planar video image, and a planar stillimage.
 9. The communication system of claim 1, further comprising: thefirst apparatus, which is configured to transmit, to the relayapparatus, equirectangular projection video image data, wherein therelay apparatus is further configured to generate spherical video imagedata based on the equirectangular projection video image data.
 10. Thecommunication system of claim 9, further comprising: the secondapparatus, which is communicably connected with the first apparatus viathe relay apparatus, wherein the second apparatus is at least one of apersonal computer configured to reproduce the spherical video imagedata, a video conference terminal configured to reproduce planar videoimage data, and an electronic whiteboard configured to reproduce planarstill image data.
 11. A method of processing image data in acommunication system that includes a relay apparatus, communicativelycoupled to a first apparatus and a second apparatus over a network, fortransferring image data transmitted from a transmission source to atransmission destination, and a server, which controls communicationbetween the first apparatus and the second apparatus, the methodcomprising: receiving, by the relay apparatus, video image data from thefirst apparatus being the transmission source; determining, by the relayapparatus, whether the second apparatus being a transmission destinationis configured to reproduce the video image data received from the firstapparatus; based on a determination that the second apparatus is notconfigured to reproduce the video image data received from the firstapparatus, applying, by the relay apparatus, image processing to thevideo image data to generate converted image data compatible with thesecond apparatus; transferring, by the relay apparatus, the convertedimage data compatible with the second apparatus to the second apparatus;storing, in a memory of the server and for each transmission source thathas been registered, information on a type of image that istransmittable by the transmission source and reproducible by thetransmission destination; receiving, by the server, a request fortransmitting image data from the second apparatus, the request includingan identifier of the first apparatus being the transmission source andan identifier of the second apparatus being the transmissiondestination; updating, by the server, the information on the type ofimage stored for the first apparatus to include a type of image that isreproducible by the second apparatus, as the type of image that istransmittable by the first apparatus and reproducible by the secondapparatus; transmitting, by the server, a transfer request fortransferring the video image data to the relay apparatus, the transferrequest including information on the type of image that is transmittableby the first apparatus and reproducible by the second apparatus; anddetermining, by the relay apparatus, whether the second apparatus isconfigured to reproduce the video image data, based on the informationon the type of image that is received from the server.
 12. The method ofclaim 11, wherein in a case that the video image data received from thefirst apparatus is spherical video image data, and the determinationindicates that the second apparatus is not configured to reproduce thespherical video image data, the method further comprises: determiningwhether the second apparatus is configured to reproduce planar videoimage data; and based on a determination that the second apparatus isconfigured to reproduce planar video image data, generating planar videoimage data corresponding to a specific area of the spherical video imagedata, the specific area being defined by a specific imaging directionand a specific angle of view; and transferring the planar video imagedata to the second apparatus.
 13. The method of claim 12, wherein in acase that the determination indicates that the second apparatus is notconfigured to reproduce planar video image data, the method furthercomprises: generating planar still image data corresponding to aspecific area of the spherical video image data, the specific area beingdefined by a specific imaging direction and a specific angle of view;and transferring the planar still image data to the second apparatus.